OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

naemono · 2023-01-05T16:17:23Z

This adds a Buildkite pipeline for release operations pertaining to OperatorHub/Red Hat.
This also augments our existing OpertaorHub CLI and adds new commands/sub-commands.

~~What's been manually tested:~~

~~Full set of operatorhub release operations with a BC build of Operator.~~

~~In progress:~~

~~Currently testing Buildkite integration (will fully test during actual release).~~

^^ All fully tested during release process for 2.6.x.

Adding vault option for easier usage. move vault to container command move single bundle command to push + publish commands make create-pr separate command

adding prefight additional fixes

Fix issues in create-pr tooling.

code cleanup

un-export funcs Adjust some comments

removal of 'all' command. Removing preflight

move container prerune to persistent to ensure sub-commands also have it run cleanup readme.

flags validation update. attempting to hide root flags on sub-commands.

Cleanup readme

Make vault secrets default properly.

Update readme Update some env var comments.

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Update example command for generate-manifests. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

thbkrkr

Maybe I miss something but as discussed in #6281 (comment), I think we shouldn't duplicate what is defined in ./config.yaml (OHUB_CONF). This configuration file is the source of truth that lives in the codebase and is updated only through pull requests.
This means that all the following flags / env vars should disappear:

OHUB_SUPPORTED_OPENSHIFT_VERSIONS
OHUB_STACK_VERSION
OHUB_PREV_VERSION
OHUB_TAG
OHUB_BRANCH

It seems to me that having to adjust them is painful and above all prone to error.

.buildkite/pipeline-release-operatorhub.yml

.buildkite/scripts/operatorhub/operations.sh

.buildkite/pipeline-release-operatorhub.yml

hack/operatorhub/internal/container/container.go

.buildkite/scripts/operatorhub/operations.sh

thbkrkr · 2023-02-10T14:00:17Z

hack/operatorhub/README.md

+# If wanting to override newVersion|prevVersion|stackVersion variables in config.yaml
+./bin/operatorhub generate-manifests -c config.yaml -y ../../config/crds.yaml -y ./../config/operator.yaml -s '8.6.0' -V '2.5.0' -t '2.6.0'


What is the use case that would require overriding the values defined in config.yaml?

I think I was wanting to ensure that the settings were absolutely correct when running in an automated fashion, but since the typical use case for this is to run this from a branch that already has config.yaml updated appropriately, then we can likely do without these, agreed. They've been removed.

thbkrkr · 2023-02-10T14:14:00Z

.buildkite/scripts/operatorhub/operations.sh

+    cd hack/operatorhub
+    /usr/local/bin/operatorhub generate-manifests --prev-version="$OHUB_PREV_VERSION" --stack-version="$OHUB_STACK_VERSION" --yaml-manifest=../../config/crds.yaml --yaml-manifest=../../config/operator.yaml
+    /usr/local/bin/operatorhub bundle generate --dir="$(pwd)"
+    /usr/local/bin/operatorhub bundle create-pr --dir="$(pwd)"


I would have kept these release commands in the pipeline and just use this script for the preflight commands.

I figured since I was cleaning up the pipeline, I might as well keep it as clean as possible, hence moving all of this to a script. Lmk if you need me to move it back.

thbkrkr · 2023-02-10T14:56:00Z

hack/operatorhub/cmd/buildkite/buildkite.go

+	cmd := &cobra.Command{
+		Use:          "buildkite",
+		Short:        "Start operatorhub release operation within Buildkite",
+		Long:         "Start operatorhub release operation within Buildkite.",


I think

curl -XPOST "https://api.buildkite.com/v2/organizations/elastic/pipelines/cloud-on-k8s-operator-release-operatorhub/builds" \ -H "Authorization: Bearer $BK_TOKEN" \ -d '{ "commit": "2.6.1", "branch": "2.6" }'

could have been enough.

We discussed this privately, and decided that opening #6408 would be beneficial.

hack/operatorhub/templates/package.tpl

Try and chmod the binary prior to uploading as bk artifact. Add headers to bash scripts. Silence curl. Adjust preflight command in script to actually call preflight. Use io instead of ioutil. Remove unused vars. Use http.NewRequestWithContext instead of http.NewRequest. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

ApiKey => APIKey per linter. README updates. Use io instead of ioutil. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

naemono · 2023-02-13T15:14:41Z

Maybe I miss something but as discussed in #6281 (comment), I think we shouldn't duplicate what is defined in ./config.yaml (OHUB_CONF). This configuration file is the source of truth that lives in the codebase and is updated only through pull requests. This means that all the following flags / env vars should disappear:

OHUB_SUPPORTED_OPENSHIFT_VERSIONS

OHUB_STACK_VERSION

OHUB_PREV_VERSION

OHUB_TAG

OHUB_BRANCH

It seems to me that having to adjust them is painful and above all prone to error.

@thbkrkr I've removed prev/stack, but removing the others is a much bigger change as, for example, tag is used throughout the tool. I'll get these removed/updated soon.

Remove prev/stack version flags, and use values in config file. Remove need for OHUB_BRANCH. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

naemono · 2023-02-14T15:42:28Z

@thbkrkr changes made to remove need for

OHUB_STACK_VERSION
OHUB_PREV_VERSION
OHUB_TAG
OHUB_BRANCH

Unfortunately OHUB_SUPPORTED_OPENSHIFT_VERSIONS is required, as there's no setting for it in the configuration file. If you want, I can certainly add that as a new option there and remove the cli flag. LMK.

thbkrkr · 2023-02-14T17:01:22Z

Unfortunately OHUB_SUPPORTED_OPENSHIFT_VERSIONS is required, as there's no setting for it in the configuration file. If you want, I can certainly add that as a new option there and remove the cli flag. LMK.

I think it's good to have all the configuration in the source files, so I feel like saying yes please.
Should it be more something like minSupportedOpenShiftVersion as we don't want the range here?

We already have the range in the template that we could half-generate from it or accept to have the info duplicated (we already have the minKubeVersion duplicated in the template).

cloud-on-k8s/hack/operatorhub/templates/csv.tpl

Line 270 in d8d38f5

* OpenShift 4.7-4.11

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

naemono · 2023-02-14T18:13:35Z

We already have the range in the template that we could half-generate from it or accept to have the info duplicated (we already have the minKubeVersion duplicated in the template).

I've kept it simple, and just added it to the configuration file, and updated the validation to ensure that it exists, and is not a range.

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

This reverts commit 36196b4.

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

thbkrkr

LGTM!

The vault bits could benefit from #6418 but we shouldn't block for that.

naemono added 27 commits January 12, 2022 07:38

Add sub-commands: bundle, container, operatorhub

b5cad6a

Adding vault option for easier usage. move vault to container command move single bundle command to push + publish commands make create-pr separate command

wip

52be31f

Minimally functional build with preflight.

e7148ec

Merge branch 'main' into 258-simpify-redhat-publishing

fafc063

wip

5ef2a3a

wip

2e19acc

removing pterm

4f112dd

adding prefight additional fixes

move -enable-vault to global

87ce09a

Vault work to automatically retrieve sensitive data.

59da0dc

wip

f9a1c46

wip

f35c5c7

wip

13ff70b

Move some flags to persistent flags in parent commands.

db4006d

Fix issues in create-pr tooling.

docs update

85c6aaf

code cleanup

code cleanup

8007980

rename operatorhu package

24dbab3

un-export funcs Adjust some comments

Add flags and manifests packages

c788341

Cleanup and documentation

0f3eb2b

removal of 'all' command. Removing preflight

do not enable vault by default.

dc26eb7

move container prerune to persistent to ensure sub-commands also have it run cleanup readme.

readme cleanup

82ee8cf

flags validation update. attempting to hide root flags on sub-commands.

Cleanup commands

b90cca6

Cleanup readme

Move to internal instead of pkg directory.

9f02927

Cleanup and fixes during testing

bdebfcd

Update buildkite steps.

36f6a94

Make vault secrets default properly.

Add buildkite command

68bb75b

Update readme Update some env var comments.

Merge branch 'main' into 258-simpify-redhat-publishing

d99cdf9

Fix merge to main with 'go mod tidy'

0320673

naemono added >enhancement Enhancement of existing functionality exclude-from-release-notes Exclude this PR from appearing in the release notes labels Jan 5, 2023

Remove comments from pipeline, and add newline

3f73b68

naemono added 7 commits January 23, 2023 10:38

Move large commands to a script in buildkite.

7e13c98

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Update README for generate-manifests.

b92e744

Update example command for generate-manifests. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Move operatorhub steps to a custom pipline yaml

ab22eed

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Merge branch 'main' into 258-simpify-redhat-publishing

b7f9859

Go mod tidy

9368f9c

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Fix shellcheck issues

d1ec9c6

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Merge branch 'main' into 258-simpify-redhat-publishing

dfb1fe9

naemono mentioned this pull request Feb 6, 2023

Move to vX.X.X tags for easier go module imports #6298

Merged

1 task

pebrc requested a review from thbkrkr February 9, 2023 14:50

thbkrkr reviewed Feb 10, 2023

View reviewed changes

hack/operatorhub/templates/package.tpl Outdated Show resolved Hide resolved

naemono and others added 4 commits February 10, 2023 10:02

Make script executable.

3b4ad17

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Stack/Prev version flags removed for ohub cmd.

73f35a0

ApiKey => APIKey per linter. README updates. Use io instead of ioutil. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Keep the 'v' in the ohub package template.

da31c9c

Co-authored-by: Thibault Richard <thbkrkr@users.noreply.github.com>

Remove need for cli flag --tag, just use newVersion in config file.

ce138f6

Remove prev/stack version flags, and use values in config file. Remove need for OHUB_BRANCH. Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Remove supported-openshift-versions flag

f902e09

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

naemono added 5 commits February 16, 2023 10:52

Merge branch 'main' into 258-simpify-redhat-publishing

c50792a

Add back fake failing tests.

36196b4

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

Revert "Add back fake failing tests."

2002363

This reverts commit 36196b4.

Merge branch 'main' into 258-simpify-redhat-publishing

2ee5b58

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

go mod tidy

2db9415

Signed-off-by: Michael Montgomery <mmontg1@gmail.com>

thbkrkr approved these changes Feb 23, 2023

View reviewed changes

Merge branch 'main' into 258-simpify-redhat-publishing

5134424

naemono merged commit 6e5d6e1 into elastic:main Feb 27, 2023

naemono deleted the 258-simpify-redhat-publishing branch February 27, 2023 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

naemono commented Jan 5, 2023 •

edited

Loading

thbkrkr left a comment

thbkrkr Feb 10, 2023

naemono Feb 13, 2023

thbkrkr Feb 10, 2023

naemono Feb 13, 2023

thbkrkr Feb 10, 2023

naemono Feb 13, 2023

naemono commented Feb 13, 2023

naemono commented Feb 14, 2023

thbkrkr commented Feb 14, 2023

naemono commented Feb 14, 2023

thbkrkr left a comment

		# If wanting to override newVersion\|prevVersion\|stackVersion variables in config.yaml
		./bin/operatorhub generate-manifests -c config.yaml -y ../../config/crds.yaml -y ./../config/operator.yaml -s '8.6.0' -V '2.5.0' -t '2.6.0'

OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

OperatorHub Release CLI / Moving OperatorHub release to Buildkite #6281

Conversation

naemono commented Jan 5, 2023 • edited Loading

thbkrkr left a comment

Choose a reason for hiding this comment

thbkrkr Feb 10, 2023

Choose a reason for hiding this comment

naemono Feb 13, 2023

Choose a reason for hiding this comment

thbkrkr Feb 10, 2023

Choose a reason for hiding this comment

naemono Feb 13, 2023

Choose a reason for hiding this comment

thbkrkr Feb 10, 2023

Choose a reason for hiding this comment

naemono Feb 13, 2023

Choose a reason for hiding this comment

naemono commented Feb 13, 2023

naemono commented Feb 14, 2023

thbkrkr commented Feb 14, 2023

naemono commented Feb 14, 2023

thbkrkr left a comment

Choose a reason for hiding this comment

naemono commented Jan 5, 2023 •

edited

Loading